fix: preserve Unicode when clipping file lines - #1960

…#1948)

Build on the extracted Unicode clipping, shared image MIME guard, and streamed path fixes from PRs Zoo-Code-Org#1960, Zoo-Code-Org#1961, and Zoo-Code-Org#1962. Preserve their latest regression coverage while composing reader strategies, descriptor-bound access, approval, cancellation, and validated text/document results.

…#1948)

Build on the extracted Unicode clipping, shared image MIME guard, and streamed path fixes from PRs Zoo-Code-Org#1960, Zoo-Code-Org#1961, and Zoo-Code-Org#1962. Preserve their latest regression coverage while composing reader strategies, descriptor-bound access, approval, cancellation, and validated text/document results.

…#1948)

Build on the extracted Unicode clipping, shared image MIME guard, and streamed path fixes from PRs Zoo-Code-Org#1960, Zoo-Code-Org#1961, and Zoo-Code-Org#1962. Preserve their latest regression coverage while composing reader strategies, descriptor-bound access, approval, cancellation, and validated text/document results.

…dows

This makes content past 2000 characters on a single line readable at all. It
previously had no address any parameter could express, leaving command execution
as the only way to reach it.

`read_file` truncated every line at 2000 characters, and content past that point
was completely unreachable. Not merely inconvenient, but unreachable: no
combination of parameters could return it.

The reason is that `read_file`'s `offset` and `limit` select *which lines* to
return. Neither has ever addressed a position *within* a line, and there was no
`column_offset` or equivalent. So for a 5000-character line, the agent could ask
for "line 2", receive the first 1997 characters followed by a bare `...`, and
then had no way to ask for anything further. The remaining 3000 characters had
no address the tool could express.

The practical effect: any file whose lines are long was partly invisible.
Single-line JSON, minified JS and CSS, CSV exports, HTML email bodies, long log
lines. The agent's only recourse was `execute_command`, shelling out to `sed`,
`cut` or `tail`, which requires command approval, pollutes the transcript, and
fails on Windows where those tools are usually absent. Any text editor or `less`
displays this content without difficulty; the agent could not.

Worse, the `...` marker said nothing about *how much* was missing, so the agent
could not even tell it was looking at a partial line worth investigating.

Add a third mode to `read_file`, alongside the existing `slice` (read lines
N..M) and `indentation` (read the code block around line N):

    read_file { path, mode: "bytes_as_utf8", bytes_as_utf8: { offset, limit } }

This reads a window of raw bytes at a byte offset and decodes it as UTF-8,
giving every position in a file an address regardless of where line breaks fall.
It reads with a positioned `fs.open`/`read` instead of loading the file, so a
file much larger than the window, or larger than memory, can still be paged
through a window at a time.

The window lives in its own `bytes_as_utf8: { offset, limit }` object rather
than reusing the top-level `offset`/`limit`, which remain 1-based *line* numbers
and are documented as ignored in this mode. Silently changing their unit based
on `mode` is how a caller ends up reading byte 1500 when it meant line 1500.

Truncation markers now state exactly what was dropped and how to get it,
replacing the bare `...`:

      2 | {"payload":"AAAA...[+1026 bytes omitted, starting at byte offset 5000
    up to and including byte offset 6025]

    Note: 1 line exceeded the 2000-byte per-line display cap (longest: 6026
    bytes). The omitted bytes have no line-based address.
    To read the omitted part of line 2: read_file with mode='bytes_as_utf8' and
    bytes_as_utf8.offset=5000.

Both endpoints are named, and both are inclusive, so acting on them needs no
arithmetic: the start offset goes directly into `bytes_as_utf8.offset`, and the
end offset says how large the remainder is without a probing read first. The
summary block repeats this once below the content, because a marker at the end
of a 2000-character line is easy to scroll past.

Because the output is *not* a faithful representation of arbitrary bytes: any
byte that is not valid UTF-8 is replaced with U+FFFD. Calling the mode `bytes`
would invite an agent to trust it for binary data it silently corrupts. The
explicit name also keeps `bytes` free for a future mode that shows bytes
verbatim, for example as hex.

**Byte offsets must be measured on the raw bytes, not on decoded text.** When
invalid bytes are decoded to U+FFFD and that text is re-encoded, each such byte
becomes three, so every offset after it shifts. Offsets derived from decoded
text would be wrong on precisely the malformed machine-generated files this
feature exists to read, and would send the follow-up byte read to the wrong
place. The file's `Buffer` is therefore passed through the read path, and line
boundaries are located by scanning it for 0x0A (via the native `Buffer.indexOf`)
instead of splitting a string.

**The per-line cap is now measured in bytes rather than characters**, because
that is the only way its cut point can be reported as a byte offset the new mode
accepts. An arbitrary byte position can fall inside a multi-byte character, so
the cut backs off to the nearest character boundary; otherwise the last visible
"character" would be a corrupt replacement glyph. Relatedly, the full 2000-byte
budget now goes to content with the marker appended outside it. Previously three
characters were reserved for `...`, so only 1997 were kept, three fewer than
documented.

Upstream Zoo-Code-Org#1960 ("fix(read-file): preserve Unicode when clipping file lines")
fixed the old character-based cut splitting a UTF-16 surrogate pair, by dropping
one more code unit when the last one kept was a high surrogate. This commit
replaces that cut, so the fix is superseded rather than kept alongside:

* Cutting at a UTF-8 sequence boundary never splits a code point, so it cannot
  split a surrogate pair either; the separate check has nothing left to catch.
* The byte-based cut point has a byte offset the marker can name. A character
  count cannot be turned into one reliably: on a file containing invalid UTF-8,
  offsets measured from the decoded text drift (see above).
* Upstream still ends the line with a bare `...`, which reads the same as a
  literal `...` in the file and says nothing about how much is missing.

The cost: the cap now counts bytes, so non-ASCII lines show fewer characters
before the cut. A line of emoji shows 500 instead of 998, CJK text about 666
characters. Neither version keeps grapheme clusters together; both only
guarantee that no code point is broken.

Upstream's `indentation-reader-unicode.spec.ts` is ported to the byte-based API
here, keeping its cases (sequences before, on and after the cap, the code points
around the surrogate range, combining marks, ZWJ sequences, and the slice and
indentation readers) with expectations re-derived for UTF-8 byte boundaries.

* Byte mode runs before the binary-file check and before any whole-file read.
  Both would defeat the point: the mode exists for files too large to hold or
  too unstructured for line reading, and an explicit byte offset is an
  unambiguous request for those bytes whatever the file's detected type.
* Reading at or past end-of-file returns an empty window rather than an error,
  so an agent paging by following reported offsets needs no special case for the
  final read.
* A multi-byte character split by the *window* edge is trimmed and reported,
  rather than decoded into a U+FFFD the agent could not distinguish from a
  replacement character genuinely present in the file. At end-of-file such a
  sequence is kept instead, since there it is real malformed data rather than an
  artefact of where the window stopped.
* Invalid bytes inside a window are counted and reported, for the same reason:
  so unexpected glyphs can be attributed to the file rather than to the read.
* The approval prompt shown to the user names bytes rather than lines, and omits
  an end offset when no limit was given, since the file may be shorter than the
  default window.

Existing `slice` and `indentation` behaviour is unchanged apart from the improved
truncation marker.

Assisted-by: Claude Opus 5 with Zoo Code

…#1948)

Build on the extracted Unicode clipping, shared image MIME guard, and streamed path fixes from PRs Zoo-Code-Org#1960, Zoo-Code-Org#1961, and Zoo-Code-Org#1962. Preserve their latest regression coverage while composing reader strategies, descriptor-bound access, approval, cancellation, and validated text/document results.